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Statistical Analysis 
for the Solomon Four-Group Design 



Gijsbert van Engelenburg 



Abstract 



The Solomon four-group design is a very useful experimental design to investigate the 
main effect of a pretest and the interaction effect of pretest and treatment. Although 
the design has been proposed half a century ago, no proper data analysis techniques 
have been available. In this paper it is described how data from the Solomon four- 
group design can be properly analyzed using maximum likelihood regression analysis. 
The results show that both the parameter estimators and the expressions for the 
standard errors are in agreement with related designs and with intuitive expectations. 
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Statistical analysis for the Solomon Four-Group Design 

Whether pretests should be administered in experimental control group designs 
is a delicate issue. On the one hand, the statistical argument is that pretests are usually 
favorable, since controlling for pretest scores will usually increase the power of the 
design by reducing residual variance. If a pretest is administered, the probability that 
small treatment effects are detected will usually increase. On the other hand, the 
methodological argument is that pretests are often unfavorable, since pretests can blur 
the real treatment effect. Two situations must be distinguished (see e.g. Campbell & 
Stanley, 1966; Cook & Campbell, 1979). First, the pretest can have an effect by itself 
by decreasing or increasing the scores of interest. For instance, administering a pretest 
may offer the subject additional exercise material, item training or a search strategy. 
Such an effect, commonly referred to as a testing effect, is a main effect of the pretest. 
Second, the pretest can make the treatment less or more effective. For instance, 
administering a pretest may arouse curiosity or it may sensitize the subject. Such an 
effect is an interaction effect of pretest and treatment. Although we will usually not 
expect that pretest main effects and interaction effects are very large (see Wilson & 
Putnam, 1982), it will be usually difficult to predict whether the statistical or the 
methodological argument prevails in a given situation. 

To investigate the role of a pretest the Solomon four-group design (Solomon, 
1949) is a useful device. Using this design the treatment effect, the pretest effect, and 
the interaction effect of pretest and treatment can be disentangled. A graphical 
depiction of the design is shown in Figure 1. Subjects are randomly (denoted by R) 
assigned to one of four groups (denoted by O, P, T, and TP), presented rowwise. 
Columnwise the experimental interventions are presented. Group P and TP get a 
pretest (X), group T and TP get the treatment (T), and all groups get a posttest (Y). 



Insert Figure 1 about here 



A striking impression from the existing literature is that the same authors who 
praise the Solomon four-group design as a very useful device, admit that the statistical 
treatment of the design is not straightforward. Oliver and Berger (1980) studied 
several pretest designs and concluded that only the Solomon four-group design 
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yielded unambiguous results. They also noted that there are many possible ways of 
analyzing the data, which is in this case another way of saying that no single proper 
analysis technique is known. Solomon's (1949) suggestions are not of much help and 
even "judged unacceptable" by Campbell and Stanley (1966, p. 25). They noted that 
”[t]here is no singular statistical procedure which makes use of all six sets of 
observations simultaneously" (p.25). Sawilovski, Kelly, Blair, and Markman (1994) 
believe that the lack of a clear statistical treatment is probably the reason that there are 
few examples of published research based on the Solomon four-group design. 

The most straightforward statistical treatment is probably to reorganize the 
posttest data according to a 2 x 2 factorial design, where the first factor is whether a 
pretest has been given or not, and the second whether a treatment has been given or 
not (Figure 2). It is clear that an ANOVA can provide estimates of the treatment 
effect, of the pretest effect, and of the interaction effect of pretest and treatment. 
However, by analyzing the data this way, not all available information is used. 
Administering pretests will, of course, result in pretest scores. Although pretest scores 
could be used to obtain better estimates, they are neglected in the ANOVA model. 
Campbell & Stanley (1966) advise to use the 2 x 2 ANOVA design and they conclude 
that ”[i]f the main and interactive effects of pretesting are negligible, it may be 
desirable to perform an analysis of covariance..." (Campbell & Stanley, 1966, p.25), 
where they refer to analyzing only the data from group P and TP to test whether an 
effect of the treatment can be found. (See also Huck & Sandler, 1973.) 



Insert Figure 2 about here 



A second obvious way to analyse the data from the Solomon four-group design 
is more or less in the spirit of the latter advice of Campbell and Stanley (1966). It is 
possible to view the Solomon four-group as a combination of two experiments. In the 
first experiment only group O and T are considered. Since no pretest was 
administered, this experiment can be treated as a posttest-only control group 
experimental design. In the second experiment only group P and TP are considered. In 
these groups a pretest was administered and this experiment can be treated as a 
pretest-posttest control group experimental design. 
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Some other attempts are made to analyse Solomon four-group data. Walton 
Braver & Braver (1988) have suggested the use of a meta-analytic approach to 
combine the results of both separate experiments. Their approach has been criticized 
by Sawilovski, Kelly, Blair, & Markman (1994), who show that the approach results 
in a lack of power and an inflation of the nominal alpha. They conclude that "further 
study is needed to resolve the data analysis problem associated with this design" 
(Sawilovski, Kelly, Blair, & Markman, 1994, p 374). Dukes, Ullman, and Stein 
(1995) made multiple comparisons between two groups at the time using a latent 
variable approach. 

In this paper a full information maximum likelihood method is presented to 
analyze all available data from the Solomon four-group design. The results will be 
compared with the results of the 2 x 2 factorial design and with the results of the 
combination of the pretest-only and the pretest-posttest designs. First, these 
commonly used approaches will be presented, then the new approach. All approaches 
will be presented in a maximum likelihood regression context, to allow simple 
comparisons between them. Before presenting the models, however, some general 
remarks are made, concerning notation, assumptions and estimation procedure. 



General remarks 
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Notation 

First some notation will be introduced. The four groups of the Solomon four- 
group design will be denoted by Q = O, P, T, TP, as indicated in Figure 2. Groups will 
be treated as nonoverlapping sets of subjects. The four groups are identified by two 
dichotomous design variables. The first design variable, 7), indicates whether subject i 
has received a treatment (T, = 1) or not (7) = 0) and, second, P, is the design variable 
which indicates whether for subject i a pretest has been administered (P, = 1) or not 
(P, = 0). 

Subjects are denoted by i = 1, 2, ..., N, where N is the total number of subjects. 
The number of subjects in group Q will be indicated by Nq. The posttest score of 
subject i will be denoted by K„ the pretest score, if available, by X,-. In this paper 
summation is always over subjects and the summation index denotes the group in 
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which summation takes place. If more than one summation index is given, summation 
takes place in all groups mentioned. Thus 

X(r,) = X(F,) + X(F,) = X(F, ) + ]►>,). (1) 

QR Q R ieQ ieR 

is the sum of the posttest scores of all subjects belonging to group Q and R. 

Further, many formulas contain sum of squares or sums of crossproducts and it 
is useful to have a shorthand notation for them. We will define the function SSTq to be 

SST e <U,V) = •£(/?, (2 ) 

Q 



which is the "total" sum of crossproducts within a single group or, if U = V, the sum 
of squares. Further, we will define the function SSWq to be 

SSW^U.V)^^, ( 3 ) 

ieQ Q JeQ ™ q j e Q 

which is the "within" group sum of crossproducts or sum of squares. Finally, the 
function SSBq is defined as 

"Q Q Q 

which is the "between" group sum of crossproducts or sum of squares. It is well 
known and easy to verify that 

SST a (U,V) = SSWq (U,V)+ SSBq (U,V) . (5) 



Multiple indices will be used to denote summation over several groups. For example, 
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SSW QR (U,V) = SSW Q (U,V) + SSW R (U,V), (6) 

which is generally unequal to SSW QUR (U,V) . An analogous result is true for 
ssb Q ' R (u ,V) . 



Assumptions 

To allow maximum likelihood estimation, it will be assumed that subjects are 
randomly assigned to the four groups. For the sake of generality, we will not assume 
that all groupsizes are equal. Further, we will assume that error components are 
independently and normally distributed with mean zero and fixed, but unknown 
variance. It will also be assumed that the size of all experimental effects are fixed, but 
unknown. Thus, first, the size of a potential treatment effect is equal for all subjects 
who have received the experimental treatment, second, the size of a potential (main) 
effect of the pretest is also equal for all subjects who have received the pretest, and, 
third, the size of a potential interaction effect is also equal for all subjects who 
received both a pretest and a treatment. 

Further, as far as pretested subjects are considered, a linear relation is assumed 
between pretest scores and posttest scores. For notational convenience we will assume 
that pretest scores, but not posttest scores, are deviation scores from the observed 
mean. Thus, if X' t is the raw pretest score and 



X = 




( 7 ) 



then 




( 8 ) 



It follows that 




( 9 ) 



p 
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which will be useful in simplifying some of the formulas. 

Estimation 

The models presented in the remainder were all analyzed in the same way, 
according to maximum likelihood estimation. Maximum likelihood estimation for 
regression models is the relatively straightforward. Here, estimation of the parameters 
and their standard errors is described only briefly. First, the likelihood of the data is 
expressed as a function of the model parameters, given the observed data. Then, first 
derivatives of the logarithm of the likelihood function to the parameters are 
algebraically derived. The first derivatives are set equal to zero and from this set of 
equations the parameters must be solved. For the models below, this proves to be not 
too difficult. To obtain estimates of the standard errors, first the matrix of second 
derivatives of the logarithm of the likelihood function to the parameters must be 
obtained. Next, the negative of this matrix must be inverted. The matrix, however, 
proves to be relatively simple and inverting can be done algebraically by standard 
mathematics software. The inverted matrix is the variance-covariance matrix of the 
maximum likelihood parameter estimates. Standard errors can be computed by taking 
the square root of the appropriate diagonal elements. 



Commonly used methods 



The 2x2 factorial model 

If the Solomon four-group design is analysed as a 2 x 2 factorial design, the 
pretest scores are ignored. We are, of course, interested in differences between the 
four groups in the mean value of F, and the model is 

Y{ — a + bj T{ + bp P , + bpp T, Pi + /?„ (10) 



where bp, bp, and bpp are the regression coefficients associated with 7',, P t , and the 
interaction effect of T, and P„ respectively and /?, is the residual term. The coefficient 
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br is interpreted as the treatment effect, b P is interpreted as the (main) effect of the 
pretest, and b T p is interpreted as the additional effect if both a treatment and a pretest 
are given. 

Many textbooks on regression analysis or analysis of variance treat the 
estimation of parameters of a 2 x 2 factorial model. Here, maximum likelihood 
estimation is applied to allow comparisons between this model and the other models. 
If it is assumed that /?, is independently normally distributed with mean zero and 
variance a\, 

R,~N(O,0 2 s ), (11) 



then it follows that F, is independently normally distributed with mean a + b T T, + bp 
Pi+ bpp Ti Pi and variance a \ , so that the logarithm of the likelihood function is 



logL= -ylOg(2^)-iV10g(T R 



( 



Y i -a — b T T i -b p P i -b TP T i P i 



02 ) 



where summation is over all subjects. Parameter estimation is without problems and 
the resulting parameter estimates are given in Table 1. Also, the variance of the 
parameter estimates are not too difficult to find. They are also presented in Table 1. 



Insert Table 1 about here 



Using the 2x2 factorial model to analyze Solomon four-group data pretest 
scores are neglected. We may expect that parameter estimates can be improved by 
using pretest scores. Also, a reduction of the standard errors of the estimates can be 
expected. 

The posttest-onlv and pretest-posttest model 

The Solomon four-group design can be conceived as two separate experiments, 
a posttest-only control group experiment and an experiment with pretest and posttest. 
Below it is investigated what the result will be if both models are analyzed separately. 
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For the subjects in groups O and T, who did not have a pretest, the model 
simply is 

Yi = a + brTi + Ri, forieOuT. (13) 



For this model parameter estimation is very much analogous to the estimation for the 
2x2 factorial design. It follows that F, is independently normally distributed with 
mean a + brTi and variance o\, so that the logarithm of the likelihood function is 



logL = 



N_o +Nj. 
2 



log(2 a) - (N 0 +N T )logc R -W 

2 O.T 



Yj—a—b T T i 



V 



(14) 



The maximum likelihood parameter estimators are presented in Table 2, as are the 
associated variance estimates. 



Insert Table 2 about here 



The subjects in groups P and TP, however, did have a pretest and pretest scores 
can be entered into the regression analysis to reduce the residual variance. Since all 
subjects did have a pretest the main effect of the pretest can not be separated from the 
intercept. This combined parameter will be denoted by a', thus 

a' - a + bp. ( 15 ) 

Also, in this model an effect of the treatment cannot be separated from the interaction 
effect of pretest and treatment. This combined effect will be denoted by bj’, thus 



bj' — bj + bjp. 



( 16 ) 



The model, then, is 
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Y i = a' + b T 'T i + bxX i + E i , 



foriePuPT, (17) 



where, b x is the regression coefficient associated with X,. The error term £, will 
usually be smaller than the residual terms i?, in (13), because X, is now entered into 
the model. 

To find the parameters of this model, it is assumed that Et is independently 
normally distributed with mean zero and variance cr^ , 

E;~ N(0,<7 2 e ). ( 18) 



It follows that Yi is independently normally distributed with mean a' + b T ' 7) + b x X, 
and variance and the logarithm of the likelihood function is 



logL = 



N P +N. 



— log(2tf)-(iV p +N TP )\oga E -I V 



' P,TP 



Y i -a^T]-b 2L X L 

Or, 



\2 



(19) 



and, again, parameters and standard errors can be obtained in the same way. In Table 
3 parameter estimates and associated variance estimates are presented. 



Insert Table 3 about here 



In this approach not all parameters of interest are estimated explicitly. However, 
the remaining parameters can be obtained by combining the results of Table 2 and 3. 
Because of (15) an estimate of bp can be obtained by 




Ir, 

__P P o 

N P N 0 



( 20 ) 



and in the same way, because of (16) an estimate of b T p can be obtained by 
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. . . I J'.-ixI*, I* -*,!*■ In In 

b TP =b' T -b T =^- 



TP 



N t 



N P 






AL 



( 21 ) 



These results have not been stated explicitly before, although many researchers 
probably had the intuitive notion that combining the results of both designs gave 
insight about the pretest (main) effect and the interaction effect of pretest and 
treatment. An important question, however, remains. Although the estimators are 
maximum likelihood estimators, it may be doubted whether they are efficient. Limited 
information estimators, which result if not all available data are analyzed 
simultaneously, may be less efficient than full information estimators. The variance of 

* A 

b P and the variance of b^, will not be derived here. After the new approach is 
presented, a remark regarding their respective variances will be made. 

There are two main reasons to use pretest scores, and both can be illustrated 
with the current parameter estimates at hand. First, using pretest scores may correct 
for sampling fluctuations. What we see in Table 3 is that some of the parameter 
estimates are not simply the mean of the posttest scores within a certain group, but 
that the mean score is corrected for pretest differences. Second, using pretest scores 
will usually result in a smaller residual term and thereby enhancing the power of the 
significance tests of the parameters. Indeed, from d E it is clear that the sum of 

squares of posttest scores corrected for pretest differences will be usually much 
smaller than simply the sums of squares. 



New method 



Model 

The main flaws of the previous approach results from analyzing both models 
separately. Below is presented how the data from all four groups, including pretest 
data, can be analyzed simultaneously. If pretest scores were available for all subjects, 
the model is defined as 




Y, = a + bx Xj + brTj+ bp Pi + bpp T, P, + Ei, 



( 22 ) 
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To avoid the problem of the structurally missing pretest scores, for. each of the four 
groups a separate model will be derived from (22). First, for the subjects who did have 
both a pretest and a treatment the model can be written as 

Yi — a + bx Xi + br + bp + bpp + Ei, for i e TP. (23) 

Second, for the subjects who did have a pretest, but no treatment the model is 

T, = a + b x Xi + b P + Ei, for i e P. (24) 

If no pretest is administered, and thus, no pretest scores are available, the variation 
that Xj accounts for is added to the residual variation. If the residual part is denoted by 
Ri, the model for the subjects that did have a treatment but no pretest can be written as 

Yi = a + bp + Ri, for is T. (25) 

and for the subjects that had neither a pretest nor a treatment the model is 

Yi = a + Ri, for i e O. (26) 

Conceptually, the residual part /?,. can be decomposed into two parts, the "real" error 
term and a term that is related to pre-experimental differences, 

Ri = b x Xi + Ei, (27) 

although Xj is unobserved for these subjects. Note that, although a separate model is 
formulated for. each group, they will be analyzed simultaneously. 
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Parameter estimation 

Since the parameter estimation for this new model has not been described 
before, a more detailed desciption will be given. From (23) to (26) and the assumption 
of independently, normally distributed error terms, the distribution of y, can be found. 
For each group it follows that y, is normally distributed, with means and standard 
deviation depending on the group, 



/(*,) = - 



t r 42k 



exp 



y,-« 



v 



for i e O, (28) 



f(Y i ) = 



o r 42k 



exp- 



1 f y, -a-b., 



' 



for i e T, (29) 



f(Y,) = 



a E 42 n 



exp- 



Y t -a -b x X i -b p 



\ 2 



for i e P, (30) 



and 



f(Y,) = 



o E 42ii 



exp 



_ 1 


(Y i -a-b x X l -b T -b P -b TP ] 


2 ' 


2 


\ E ) 





, for i e TP. (31) 



Since the likelihood of the parameters given the observed data is 

N 

% jb-p p jbjp , cx r , cx ^ I = (32) 



the logarithm of the likelihood, written in terms separately for each group, is 
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l°gL=Y, 



, 1.11 

log -7r= + log 

V2 k a R 2 



' 2 L-*' 
, 



, 1 , 1 1 
log-== + log - 

V 2n a R 2 



(v . t V'' 



Y, -a -b T 



, 1,11 

log -7=^ + log - 

V2 n a E 2 



Y,-a-b x X, -b p 



\*\ 



♦X 

TP 



, 1,11 

log -7= + log - 

V2 n a E 2 



Y l -a-b x X l -b T -b P -b TP ) 



2 \ 



(33) 



This can be simplified to 



logL= - N log 42k - (N 0 + N T )\oga R ~(N P + N TP )\oga t 



1 



O.T 

o +b D 



YV 



E PTP ° R O V R T 



[ a +bp +bj + b TP Y'y b x v V 2 b x (a+b p ) 



' E p 



TP ™ E PJP 



Y *. 



_ (° + bp + br + hy p ) Y 1 v . bx V V V AT a 2 AT (° + b T ) 

2 2 * A i + 2 Z**i A i ™ O T~T - "t — — 

"e TP o E p TP 2 a R 2(7 B 



_ (o+fef) 2 
iy p 2 tp 

2a t, 



p > _ + bj^ + bj p ) 



2a; 



(34) 



The first derivatives are presented in Appendix 1. Setting the equations equal to 
zero results in the parameter estimators in Table 4. This exercise, although not 
difficult is somewhat space and time consuming. 



Insert Table 4 about here 



Standard errors 

The second derivatives are presented in Appendix 2. The negative of the matrix 
of second derivatives must be inverted to obtain the variance-covariance matrix. 
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Fortunately, the expressions for the matrix of second derivatives are relatively simple 
and since both the second derivatives to o E and any other parameter and the second 
derivatives to Or and any other parameter are all zero, only a 5 x 5 matrix has to be 
inverted. Algebraic invertion is not problematic using standard mathematics software. 
The expressions for the variances of the parameter estimates are presented in Table 4, 
for the covariances in Table 5. 



Insert Table 5 about here 
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Discussion 

The investigation for a new approach was motivated by the thought that by 
using pretest scores better results would be likely than that would be obtained by 
using the 2 x 2 factorial design. This means, at least, two different things. First, since 
sampling fluctuations can play a role, it is advisable to correct for pre-experimental 
differences if possible. Second, by reducing the residual variance, standard errors of 
the estimates will be smaller and, thus, significance tests more powerfull. Below we 
will discuss whether the new model fulfils the expectations. 

First a comparison with the 2 x 2 factorial model will be made. If the estimators 
of a and b T for the new model (see Table 4) are compared with the estimators for the 
factorial model (see Table 1) it is clear that both the estimators and the standard errors 
are exactly the same for both models. Undoubtly, this has to do with the fact that these 
estimators are only based on the data of the groups that were not pretested. But, for b P 
and b T p there are differences. The estimators show that the posttest scores in groups P 
and TP are indeed corrected for pretest differences. Variance estimates are changed 
too. Some Or terms are replaced by Or terms and, since <Je will usually be smaller, 
often much smaller, than Or, this indeed indicates that the variance estimates are 
shrunken. However, this is not the whole story. A component consisting of the ratio of 
“between” sums of squares of X, and “within” sums of squares of X,- is added to the Oe 
terms as a kind of penalty. Usually, this ratio will be very small, but it reminds us that 
adding a pretest may even work out worse in extreme cases. If X, is totally unrelated 
to F„ Oe will be equal to Or and the result is that the variance estimates of bp and bpp 
will be even larger for the new model than for the factorial model. 

Comparison to the combination of the posttest-only and the pretest-posttest 
model yield a different picture. If the estimates of a and b T for the new model (see 
Table 4) are compared with the estimates for the combination of the pretest-only 
model (see Table 2) and the pretest-posttest model (see Table 3) it is clear that, again 
for a and bp, there is full agreement between the models. As has been shown in (20) 
and (21) bp and bpp can be estimated by combining the results of the pretest-only 
model and the pretest-posttest model. It is clear that they are exactly equal to the 
parameter estimates of the new model. However, for the new model standard errors 
were directly obtained, which was not the case for the combination of the pretest-only 
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model and the pretest-posttest model. It is interesting to see that Var(b p ) = 

Var(a ) +Var(a) and that Var(b TP ) = Var(b ' T ) +Var(b T ) . In retrospect, this is a 
plausible result. This result could have been derived from (20) and (21) by realizing 
that Cov(a',a) = 0 and Cov(b' T ,b T ) = 0. 

In this study it has been assumed that both E, and R, are normally distributed. 
This implies that X, is also normally distributed (see (27)). These assumpties allow 
maximum likelihood estimation. However, the assumptions may not allways be 
realistic and, therefore, in some cases least squares estimation should be preferred. 
There is another reason to prefer least squares estimation. It is well known that the 
maximum likelihood estimates of the residual variance are somewhat biased, although 
the bias reduces with larger sample size. Maximum likelihood estimates of the 
residual variance can be corrected to remove the bias by adjusting the number of 
degrees of freedom and in that case they are equal to least squares estimates. Although 
not investigated, a suggestion is to remove the bias of <J R by dividing by N 0 + N T - 2 
instead of by N 0 + N T and to remove the bias of 6 E by dividing by N P + N T p - 3 
instead of N P + N pp (see Table 4). This will probably yield the least squares estimates, 
but that topic needs further investigation. 

A more or less surprising result of this study is that the results of the new model 
could have been found by combining the results of the pretest-only model and the 
pretest-posttest model. Since those models are very well known, it is a bit striking that 
so many authors have mentioned that no proper analysis technique existed for the 
Solomon four-group design. However, the findings in this paper will hopefully result 
in more studies that implement this very useful design. 
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Table 1. Parameter estimators for the 2 x 2 factorial model. 
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Table 2. Parameter estimators for the posttest-only model. 



e 


0 


Var(6) 


a 


o 


A 1 

Or 




No 


N 0 


b T 


lr, 

T O 


<*R + °l 


I 


n t n q 


N 0 N t 


ST 


jSSW or (Y ,Y) 


A 2 

Or 


U R 


If N 0 + N T 


2(N 0 +N t ) 



ERIC 



23 



Table 3. Parameter estimators for the pretest-posttest model. 
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Table 4. Parameter estimators for the new model. 
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Table 5. Covariances for the new model. 
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Figure 1. A graphical depiction of the Solomon four-group design. 
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Figure 2. The Solomon four-group design as a 2 x 2 factorial design. 
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Appendix 1: First derivatives 
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Appendix 2: Second derivatives evaluated at the maximum 
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